A new method for testing dialogue systems based on simulations of real-world conditions

This paper presents a new method for testing dialogue systems using a variety of real-world conditions simulated in lab. The method is based on the use of an additional dialogue system, called simulator, designed to behave as users interacting with the dialogue system to test. The behavior of the simulator is decided from diverse scenarios that represent user goals. The simulator tries to achieve the goals in the scenarios during the interaction with the dialogue system. We applied the method to test in noise conditions a dialogue system under development in our lab, considering white and babble noise. The method allowed to find out errors in the recognizer and in the strategy the system uses to handle user confirmations. Using the method, we tested the behavior of the system in terms of task completion, taking into account a VTS noise compensation technique. The experiments show that, if the VTS technique is not used, the average task completion is 31,43% for the white noise and 29,64% for the babble noise. If the VTS technique is used, the average task completion increases to 57,35% for the white noise and to 54,44% for the babble noise.